Recently, the anchor acceleration, an acceleration mechanism distinct from Nes-terov's, has been discovered for minimax optimization and fixed-point problems,

Extrapolation methods use the last few iterates of an optimization algorithm to produce a better estimate of the optimum. They were shown to achieve optimal convergence rates in a deterministic setting using simple gradient iterates. Here, we study extrapolation methods in a stochastic setting, where the iterates are produced by either a simple or an accelerated stochastic gradient algorithm.

We describe a convergence acceleration technique for generic optimization problems.

Neural Information Processing Systems http://nips.cc/

In this section, we fill in the missing details for proving Theorem 1, including a statement of the concentration bound used to establish Lemma 2, and a proof for Lemma 3. We first provide some

Stochastic gradient descent (SGD) gives an optimal convergence rate when minimizing convex stochastic objectives f (x). However, in terms of making the gradients small, the original SGD does not give an optimal rate, even when f (x) is convex.

Thanks to their scalability and low memory requirements, first-order methods are especially popular in this setting (Bottou et al.,

While our presentation focuses on this finite-sum structure, most of our convergence results can easily be adapted to the general stochastic setting (see App. D).